{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Validating and Importing User-Item-Interaction Data <a class=\"anchor\" id=\"top\"></a>\n",
    "\n",
    "In this notebook, you will choose a dataset and prepare it for use with Amazon Personalize.\n",
    "\n",
    "1. [Introduction](#intro)\n",
    "1. [Choose a dataset or data source](#source)\n",
    "1. [Prepare your data](#prepare)\n",
    "1. [Create dataset groups and the interactions dataset](#group_dataset)\n",
    "1. [Configure an S3 bucket and an IAM role](#bucket_role)\n",
    "1. [Import the interactions data](#import)\n",
    "\n",
    "## Introduction <a class=\"anchor\" id=\"intro\"></a>\n",
    "\n",
    "For the most part, the algorithms in Amazon Personalize (called recipes) look to solve different tasks, explained here:\n",
    "\n",
    "1. **HRNN & HRNN-Metadata** - Recommends items based on previous user interactions with items.\n",
    "1. **HRNN-Coldstart** - Recommends new items for which interaction data is not yet available.\n",
    "1. **Personalized-Ranking** - Takes a collection of items and then orders them in probable order of interest using an HRNN-like approach.\n",
    "1. **SIMS (Similar Items)** - Given one item, recommends other items also interacted with by users.\n",
    "1. **Popularity-Count** - Recommends the most popular items, if HRNN or HRNN-Metadata do not have an answer - this is returned by default.\n",
    "\n",
    "No matter the use case, the algorithms all share a base of learning on user-item-interaction data which is defined by 3 core attributes:\n",
    "\n",
    "1. **UserID** - The user who interacted\n",
    "1. **ItemID** - The item the user interacted with\n",
    "1. **Timestamp** - The time at which the interaction occurred\n",
    "\n",
    "We also support event types and event values defined by:\n",
    "\n",
    "1. **Event Type** - Categorical label of an event (browse, purchased, rated, etc).\n",
    "1. **Event Value** - A value corresponding to the event type that occurred. Generally speaking, we look for normalized values between 0 and 1 over the event types. For example, if there are three phases to complete a transaction (clicked, added-to-cart, and purchased), then there would be an event_value for each phase as 0.33, 0.66, and 1.0 respectfully.\n",
    "\n",
    "The event type and event value fields are additional data which can be used to filter the data sent for training the personalization model. In this particular exercise we will not have an event type or event value. \n",
    "\n",
    "## Choose a dataset or data source <a class=\"anchor\" id=\"source\"></a>\n",
    "[Back to top](#top)\n",
    "\n",
    "As we mentioned, the user-item-iteraction data is key for getting started with the service. This means we need to look for use cases that generate that kind of data, a few common examples are:\n",
    "\n",
    "1. Video-on-demand applications\n",
    "1. E-commerce platforms\n",
    "1. Social media aggregators / platforms\n",
    "\n",
    "There are a few guidelines for scoping a problem suitable for Personalize. We recommend the values below as a starting point, although the [official limits](https://docs.aws.amazon.com/personalize/latest/dg/limits.html) lie a little lower.\n",
    "\n",
    "* Authenticated users\n",
    "* At least 50 unique users\n",
    "* At least 100 unique items\n",
    "* At least 2 dozen interactions for each user \n",
    "\n",
    "Most of the time this is easily attainable, and if you are low in one category, you can often make up for it by having a larger number in another category.\n",
    "\n",
    "Generally speaking your data will not arrive in a perfect form for Personalize, and will take some modification to be structured correctly. This notebook looks to guide you through all of that. \n",
    "\n",
    "To begin with, we are going to use the [Last.FM](https://grouplens.org/datasets/hetrec-2011/) dataset. These are records of the music listening behavior of its users. The data fits our guidelines with a large number for users, items, and interactions.\n",
    "\n",
    "First, you will download the dataset and unzip it in a new folder using the code below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dir = \"poc_data\"\n",
    "!mkdir $data_dir\n",
    "!cd $data_dir && wget http://files.grouplens.org/datasets/hetrec2011/hetrec2011-lastfm-2k.zip\n",
    "!cd $data_dir && unzip hetrec2011-lastfm-2k.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a look at the data files you have downloaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls $data_dir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At present not much is known about the data other than we seem to have many .dat files and a README. Opening the README will tell us about the overall structure of this data. This is a step you probably can skip with custom data unless the data source is coming from an external team."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pygmentize poc_data/readme.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the README, we can see that there are multiple interaction types in this dataset. Interactions between users marking each other as friends, interactions from users listening to artists, and interactions from tags assigned to users and artists.\n",
    "\n",
    "In this case, we are focusing on the users, the artists, and the listening interactions. We have 1892 users, 17632 artists (our items in this case), and 92834 user-listened artist interactions. This is more than enough for us to get started with Personalize.\n",
    "\n",
    "Continue reading through the README to get to the `Files` section. Most of the files in the dataset are not relevant to us, but the `users_artists.dat` file looks promising. The `Data format` section of the README provides more details on the contents of the file. This is where we encounter our first problem.\n",
    "\n",
    "| userID | artistID | weight  |\n",
    "|--------|----------|---------|\n",
    "| 2      | 51       | 13883   |\n",
    "\n",
    "Although there is interaction data between users and the artists they are listening to, these interactions are stored as weights instead of timestamps. We need user-item-timestamp interaction data for Amazon Personalize. \n",
    "\n",
    "If you take another look at the files in the dataset, you should see that `users_taggedartists-timestamps.dat` does contain timestamp data. So what if we use tagging behavior as our interaction data, instead of listening behavior? Can we assume that a user tagging an artist is an indication of positive sentiment? Normally, you would discuss with your customer, or someone who has domain knowledge, to understand if this interaction is suitable for the use case you want to solve. For now, we will assume that tagging behavior is suitable for our needs. \n",
    "\n",
    "The schema for the `user_taggedartists-timestamps.dat` is:\n",
    "\n",
    "| userID | artistID | tagID | timestamp     |\n",
    "|--------|----------|-------|---------------|\n",
    "| 2      | 52       | 13    | 1238536800000 |\n",
    "\n",
    "If we remove the `tagID` attribute, we have exactly the format we need for Amazon Personalize."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare your data <a class=\"anchor\" id=\"prepare\"></a>\n",
    "[Back to top](#top)\n",
    "\n",
    "The next thing to be done is to load the data and confirm the data is in a good state, then save it to a CSV where it is ready to be used with Amazon Personalize.\n",
    "\n",
    "To get started, import a collection of Python libraries commonly used in data science."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "from time import sleep\n",
    "import json\n",
    "from datetime import datetime\n",
    "\n",
    "import boto3\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next,open the data file and take a look at the first several rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "original_data = pd.read_csv(data_dir + '/user_taggedartists-timestamps.dat')\n",
    "original_data.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clearly the data did not load correctly. The default delimiter for CSV (comma-separated value) files is a comma (`,`), but in this case the file was saved with tab (`\\t`) delimiters. Can you edit the code above to use the correct delimiter?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !ACTION REQUIRED!\n",
    "# Copy the code from the cell above and fix the delimiter issue"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's better. Now that the data has been successfully loaded into memory, let's extract some additional information. First, calculate some basic statistics from the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "original_data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This shows that we have a good range of values for `userID` and `artistID`. Next, it is always a good idea to confirm the data format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "original_data.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From this, you can see that there are a total of 186,479 entries in the dataset, with 4 columns, and each cell stored as int64 format.\n",
    "\n",
    "The int64 format is clearly suitable for `userID` and `artistID`. However, we need to diver deeper to understand the timestamps in the data. To use Amazon Personalize, you need to save timestamps in [Unix Epoch](https://en.wikipedia.org/wiki/Unix_time) format.\n",
    "\n",
    "Currently, the timestamp values are not human-readable. So let's grab an arbitrary timestamp value and figure out how to interpret it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "arb_time_stamp = original_data.iloc[50]['timestamp']\n",
    "print(arb_time_stamp)\n",
    "print(datetime.utcfromtimestamp(arb_time_stamp).strftime('%Y-%m-%d %H:%M:%S'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oops! For this particular timestamp value, the code rendered a year of 41,132. That's a bit far into the future for us, so clearly this was not the correct way to parse the data. We need a second attempt.\n",
    "\n",
    "JavaScript records time in milliseconds and this is a collection of data from a web application, so let's divide the timestamp value by 1000 before applying our code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "arb_time_stamp = arb_time_stamp/1000\n",
    "print(datetime.utcfromtimestamp(arb_time_stamp).strftime('%Y-%m-%d %H:%M:%S'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "February, 2009 feels much more realistic for our dataset. We don't need human-readable timestamps to use Amazon Personalize, but we do want the dates to be realistic, so now move forward by transforming each timestamp in the dataset away from the JavaScript milliseconds format. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "original_data.timestamp = original_data.timestamp / 1000\n",
    "original_data.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do a quick sanity check on the transformed dataset by picking an arbitrary timestamp and transforming it to a human-readable format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "arb_time_stamp = original_data.iloc[50]['timestamp']\n",
    "print(arb_time_stamp)\n",
    "print(datetime.utcfromtimestamp(arb_time_stamp).strftime('%Y-%m-%d %H:%M:%S'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This date makes sense as a timestamp, so we can continue formatting the rest of the data. Remember, the data we need is user-item-interaction data, which is `userID`, `artistID`, and `timestamp` in this case. Our dataset has an additional column, `tagID`, which can be dropped from the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interactions_df = original_data.copy()\n",
    "interactions_df = interactions_df[['userID', 'artistID', 'timestamp']]\n",
    "interactions_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After manipulating the data, always confirm if the data format has changed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interactions_df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, the timestamp column has changed from int64 to float64. So let's change the format back to int64."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interactions_df.astype({'timestamp': 'int64'}).dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Amazon Personalize has default column names for users, items, and timestamp. These default column names are `USER_ID`, `ITEM_ID`, AND `TIMESTAMP`. So the final modification to the dataset is to replace the existing column headers with the default headers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interactions_df.rename(columns = {'userID':'USER_ID', 'artistID':'ITEM_ID', \n",
    "                              'timestamp':'TIMESTAMP'}, inplace = True) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's it! At this point the data is ready to go, and we just need to save it as a CSV file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interactions_filename = \"interactions.csv\"\n",
    "interactions_df.to_csv((data_dir+\"/\"+interactions_filename), index=False, float_format='%.0f')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create dataset groups and the interactions dataset <a class=\"anchor\" id=\"group_dataset\"></a>\n",
    "[Back to top](#top)\n",
    "\n",
    "The highest level of isolation and abstraction with Amazon Personalize is a *dataset group*. Information stored within one of these dataset groups has no impact on any other dataset group or models created from one - they are completely isolated. This allows you to run many experiments and is part of how we keep your models private and fully trained only on your data. \n",
    "\n",
    "Before importing the data prepared earlier, there needs to be a dataset group and a dataset added to it that handles the interactions.\n",
    "\n",
    "Dataset groups can house the following types of information:\n",
    "\n",
    "* User-item-interactions\n",
    "* Event streams (real-time interactions)\n",
    "* User metadata\n",
    "* Item metadata\n",
    "\n",
    "Before we create the dataset group and the dataset for our interaction data, let's validate that your environment can communicate successfully with Amazon Personalize."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Configure the SDK to Personalize:\n",
    "personalize = boto3.client('personalize')\n",
    "personalize_runtime = boto3.client('personalize-runtime')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the dataset group\n",
    "\n",
    "The following cell will create a new dataset group with the name `personalize-poc-lastfm`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "create_dataset_group_response = personalize.create_dataset_group(\n",
    "    name = \"personalize-poc-lastfm\"\n",
    ")\n",
    "\n",
    "dataset_group_arn = create_dataset_group_response['datasetGroupArn']\n",
    "print(json.dumps(create_dataset_group_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we can use the dataset group, it must be active. This can take a minute or two. Execute the cell below and wait for it to show the ACTIVE status. It checks the status of the dataset group every second, up to a maximum of 3 hours."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_time = time.time() + 3*60*60 # 3 hours\n",
    "while time.time() < max_time:\n",
    "    describe_dataset_group_response = personalize.describe_dataset_group(\n",
    "        datasetGroupArn = dataset_group_arn\n",
    "    )\n",
    "    status = describe_dataset_group_response[\"datasetGroup\"][\"status\"]\n",
    "    print(\"DatasetGroup: {}\".format(status))\n",
    "    \n",
    "    if status == \"ACTIVE\" or status == \"CREATE FAILED\":\n",
    "        break\n",
    "        \n",
    "    time.sleep(60)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have a dataset group, you can create a dataset for the interaction data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the dataset\n",
    "\n",
    "First, define a schema to tell Amazon Personalize what type of dataset you are uploading. There are several reserved and mandatory keywords required in the schema, based on the type of dataset. More detailed information can be found in the [documentation](https://docs.aws.amazon.com/personalize/latest/dg/how-it-works-dataset-schema.html).\n",
    "\n",
    "Here, you will create a schema for interactions data, which needs the `USER_ID`, `ITEM_ID`, and `TIMESTAMP` fields. These must be defined in the same order in the schema as they appear in the dataset.\n",
    "\n",
    "Most of the schema is already defined below, but we've left out the `ITEM_ID` and `TIMESTAMP` fields. Can you figure out how to add them? Hint: timestamp should be of type `long`. Or take a look at the [documentation](https://docs.aws.amazon.com/personalize/latest/dg/data-prep-formatting.html) if you get stuck."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !ACTION REQUIRED!\n",
    "# Add the item ID and the timestamp fields to the schema below. The user ID has already been added as an example.\n",
    "\n",
    "interactions_schema = schema = {\n",
    "    \"type\": \"record\",\n",
    "    \"name\": \"Interactions\",\n",
    "    \"namespace\": \"com.amazonaws.personalize.schema\",\n",
    "    \"fields\": [\n",
    "        {\n",
    "            \"name\": \"USER_ID\",\n",
    "            \"type\": \"string\"\n",
    "        }\n",
    "    ],\n",
    "    \"version\": \"1.0\"\n",
    "}\n",
    "\n",
    "create_schema_response = personalize.create_schema(\n",
    "    name = \"personalize-poc-lastfm-interactions\",\n",
    "    schema = json.dumps(interactions_schema)\n",
    ")\n",
    "\n",
    "schema_arn = create_schema_response['schemaArn']\n",
    "print(json.dumps(create_schema_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With a schema created, you can create a dataset within the dataset group. Note, this does not load the data yet. This will happen a few steps later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_type = \"INTERACTIONS\"\n",
    "create_dataset_response = personalize.create_dataset(\n",
    "    name = \"personalize-poc-lastfm-ints\",\n",
    "    datasetType = dataset_type,\n",
    "    datasetGroupArn = dataset_group_arn,\n",
    "    schemaArn = schema_arn\n",
    ")\n",
    "\n",
    "interactions_dataset_arn = create_dataset_response['datasetArn']\n",
    "print(json.dumps(create_dataset_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configure an S3 bucket and an IAM  role <a class=\"anchor\" id=\"bucket_role\"></a>\n",
    "[Back to top](#top)\n",
    "\n",
    "So far, we have downloaded, manipulated, and saved the data onto the Amazon EBS instance attached to instance running this Jupyter notebook. However, Amazon Personalize will need an S3 bucket to act as the source of your data, as well as IAM roles for accessing that bucket. Let's set all of that up.\n",
    "\n",
    "Use the metadata stored on the instance underlying this Amazon SageMaker notebook, to determine the region it is operating in. If you are using a Jupyter notebook outside of Amazon SageMaker, simply define the region as a string below. The Amazon S3 bucket needs to be in the same region as the Amazon Personalize resources we have been creating so far."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('/opt/ml/metadata/resource-metadata.json') as notebook_info:\n",
    "    data = json.load(notebook_info)\n",
    "    resource_arn = data['ResourceArn']\n",
    "    region = resource_arn.split(':')[3]\n",
    "print(region)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Amazon S3 bucket names are globally unique. To create a unique bucket name, the code below will append the string `personalizepoc` to your AWS account number. Then it creates a bucket with this name in the region discovered in the previous cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3 = boto3.client('s3')\n",
    "account_id = boto3.client('sts').get_caller_identity().get('Account')\n",
    "bucket_name = account_id + \"personalizepoc\"\n",
    "print(bucket_name)\n",
    "if region != \"us-east-1\":\n",
    "    s3.create_bucket(Bucket=bucket_name, CreateBucketConfiguration={'LocationConstraint': region})\n",
    "else:\n",
    "    s3.create_bucket(Bucket=bucket_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Upload data to S3\n",
    "\n",
    "Now that your Amazon S3 bucket has been created, upload the CSV file of our user-item-interaction data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interactions_file_path = data_dir + \"/\" + interactions_filename\n",
    "boto3.Session().resource('s3').Bucket(bucket_name).Object(interactions_filename).upload_file(interactions_file_path)\n",
    "interactions_s3DataPath = \"s3://\"+bucket_name+\"/\"+interactions_filename"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set the S3 bucket policy\n",
    "Amazon Personalize needs to be able to read the contents of your S3 bucket. So add a bucket policy which allows that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "policy = {\n",
    "    \"Version\": \"2012-10-17\",\n",
    "    \"Id\": \"PersonalizeS3BucketAccessPolicy\",\n",
    "    \"Statement\": [\n",
    "        {\n",
    "            \"Sid\": \"PersonalizeS3BucketAccessPolicy\",\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Principal\": {\n",
    "                \"Service\": \"personalize.amazonaws.com\"\n",
    "            },\n",
    "            \"Action\": [\n",
    "                \"s3:*Object\",\n",
    "                \"s3:ListBucket\"\n",
    "            ],\n",
    "            \"Resource\": [\n",
    "                \"arn:aws:s3:::{}\".format(bucket_name),\n",
    "                \"arn:aws:s3:::{}/*\".format(bucket_name)\n",
    "            ]\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "\n",
    "s3.put_bucket_policy(Bucket=bucket_name, Policy=json.dumps(policy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create an IAM role\n",
    "\n",
    "Amazon Personalize needs the ability to assume roles in AWS in order to have the permissions to execute certain tasks. Let's create an IAM role and attach the required policies to it. The code below attaches very permissive policies; please use more restrictive policies for any production application."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iam = boto3.client(\"iam\")\n",
    "\n",
    "role_name = \"PersonalizeRolePOC\"\n",
    "assume_role_policy_document = {\n",
    "    \"Version\": \"2012-10-17\",\n",
    "    \"Statement\": [\n",
    "        {\n",
    "          \"Effect\": \"Allow\",\n",
    "          \"Principal\": {\n",
    "            \"Service\": \"personalize.amazonaws.com\"\n",
    "          },\n",
    "          \"Action\": \"sts:AssumeRole\"\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "\n",
    "create_role_response = iam.create_role(\n",
    "    RoleName = role_name,\n",
    "    AssumeRolePolicyDocument = json.dumps(assume_role_policy_document)\n",
    ")\n",
    "\n",
    "# AmazonPersonalizeFullAccess provides access to any S3 bucket with a name that includes \"personalize\" or \"Personalize\" \n",
    "# if you would like to use a bucket with a different name, please consider creating and attaching a new policy\n",
    "# that provides read access to your bucket or attaching the AmazonS3ReadOnlyAccess policy to the role\n",
    "policy_arn = \"arn:aws:iam::aws:policy/service-role/AmazonPersonalizeFullAccess\"\n",
    "iam.attach_role_policy(\n",
    "    RoleName = role_name,\n",
    "    PolicyArn = policy_arn\n",
    ")\n",
    "\n",
    "# Now add S3 support\n",
    "iam.attach_role_policy(\n",
    "    PolicyArn='arn:aws:iam::aws:policy/AmazonS3FullAccess',\n",
    "    RoleName=role_name\n",
    ")\n",
    "time.sleep(60) # wait for a minute to allow IAM role policy attachment to propagate\n",
    "\n",
    "role_arn = create_role_response[\"Role\"][\"Arn\"]\n",
    "print(role_arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import the interactions data <a class=\"anchor\" id=\"import\"></a>\n",
    "[Back to top](#top)\n",
    "\n",
    "Earlier you created the dataset group and dataset to house your information, so now you will execute an import job that will load the data from the S3 bucket into the Amazon Personalize dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "create_dataset_import_job_response = personalize.create_dataset_import_job(\n",
    "    jobName = \"personalize-poc-import1\",\n",
    "    datasetArn = interactions_dataset_arn,\n",
    "    dataSource = {\n",
    "        \"dataLocation\": \"s3://{}/{}\".format(bucket_name, interactions_filename)\n",
    "    },\n",
    "    roleArn = role_arn\n",
    ")\n",
    "\n",
    "dataset_import_job_arn = create_dataset_import_job_response['datasetImportJobArn']\n",
    "print(json.dumps(create_dataset_import_job_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we can use the dataset, the import job must be active. Execute the cell below and wait for it to show the ACTIVE status. It checks the status of the import job every second, up to a maximum of 3 hours.\n",
    "\n",
    "Importing the data can take some time, depending on the size of the dataset. In this workshop, the data import job should take around 15 minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "max_time = time.time() + 3*60*60 # 3 hours\n",
    "while time.time() < max_time:\n",
    "    describe_dataset_import_job_response = personalize.describe_dataset_import_job(\n",
    "        datasetImportJobArn = dataset_import_job_arn\n",
    "    )\n",
    "    status = describe_dataset_import_job_response[\"datasetImportJob\"]['status']\n",
    "    print(\"DatasetImportJob: {}\".format(status))\n",
    "    \n",
    "    if status == \"ACTIVE\" or status == \"CREATE FAILED\":\n",
    "        break\n",
    "        \n",
    "    time.sleep(60)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the dataset import is active, you are ready to start building models with SIMS, Personalized-Ranking, Popularity-Count, and HRNN. This process will continue in other notebooks. Run the cell below before moving on to store a few values for usage in the next notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store interactions_dataset_arn\n",
    "%store dataset_group_arn\n",
    "%store bucket_name\n",
    "%store role_arn\n",
    "%store role_name\n",
    "%store data_dir"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
